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ABSTRACT 

Two studies examined the effectiveness of the 
Multicultural Reading and Thinking Program (McRAT), in which teachers 
infuse higher-o der thinking strategies and performance-based 
assessment into classroom instruction. Four broad categories of 
thinking skills are included: analysis* comparison* 
inference/ interpretation* and evaluation. In the first study* 234 
McRAT students (grades 4-6) in 16 classrooms and 106 comparison 
students in 12 classrooms in 6 Arkansas schools were administered 
pre* and post-writing assessments. The second study, a follow-up of 
the first study, involved 53 second-year McRAT students who were in 
the first study and 53 first-year McRAT students enrolled in 20 McRAT 
classrooms. Assessments similar to those of the first study were 
used. Results of the first study indicated that: (1) McRAT students 
significantly exceeded comparison classes on all four thinking skills 
on the posttests; (2) far more comparison students were "off-task" 
than McRAT students on posttest measures; and (3) no statistically 
significant differences were found for boys and girls, white and 
minority students, and grades 4, 5, and 6. Results of the second 
study indicated that the differences betweon the posttest means were 
less dramatic than in the first study; however, the second-year McRAT 
students did display a slight advantage over their first-year 
counterparts. Findings of both studies suggest that McRAT students 
demonstrate significant increases in higher order thinking, and that 
these effects are independent of gender, prior achievement level or 
ethnic background* (Four tables of data are included*) (RS) 
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Multicultural Reading and Thinking Program 

(McRAT) 

Janita Hoskyn, Nancy R. Cook, Bdys S. Quellmalz, & Dan Hundfron 

This paper describes two studies reporting the results of student 
assessment for the Multicultural Reading and Thinking Program (McRAT )* McRAT 
was developed by the Arkansas Department of Education* Through a intense staff 
development process , teachers are taught to infuse higher order thinking 
strategies and performance-based assessment into classroom instruction* 
Students' thinking is assessed through essays that employ the four types of 
reasoning that are the focus of the McRAT program. Analyses of McRAT students' 
essays reveal increased ability to reason and to communicate ideas through 
writing as compared to students in non-trained teachers' classrooms. Results are 
consistent regardless of students' gender , race or prior achievement level. 
Chapter 1 students' gains are particularly striking . Additionally, there is some 
evidence that students transfer McRAT thinking strategies to other academic 
subjects and real life situations* 

McRAT embodies much of what has been learned from contemporary research 
about effective contexts for learning, staff development, literacy, and 
assessment. Further, it empowers teachers to deliver effective instruction to 
the wide range of abilities found in regular classrooms - including special 
education students, Chapter 1 and gifted students, as well as the "average" 
students. The program incorporates research on effective reading comprehension 
instruction, the interrelationship of reading, writing and thinking, effective 
staff development, the relationship of effective teaching and assessment, and the 
need for multicultural content in a increasingly more diverse nation. 

Wells (1990), Graves (1983), and Anderson (1983), among others, have 
emphasized that reading comprehension is enhanced by reading a wide range of 
content, by reading strategy instruction and by the explicit linking of writing 
and reading. Other national reports on student achievement in reading and 
writing have indicated the need for instruction which teaches students to reason 
more effectively about what they have read, to defend or elaborate on their 
ideas, and to communicate them in writing (Applebee, 1988; Langer, 1990). 



McRAT utilizes the framework of thinking skills outlined by Quellisalz 
(1985). Four broad categories of thinking skills are included: analysis , 
comparison, inference/interpretation and evaluation. This particular framework 
was selected because it addresses a common body of knowledge in philosophy and 
psychology (e.g, Bloom, 1971); it is conceptually clear; and it easily generates 
classroom assessment data (Stiggins, 1989). 

McRAT incorporates findings from research on effective staff development 
(e.g., Guskey, 1986). This body of work shows that, to change classroom 
practices, teachers need extensive training which includes modeling of the 
desired practices. Moreover, teachers need continuous support from 
administration, competent and easily accessed technical assistance, and clear 
evidence that their changes are benefitting students. McRAT* s training of 
teachers extends over two years with an initial training of nine to fourteen days 
and a follow-up of three to five days in the second year. Administrators are 
strongly encouraged to participate in the initial 3-5 days of training. The 
program also recommends that a local coordinator be named; that individual is 
expected to attend the training-of-trainers program (five additional days). 

McRAT uses literature from many cultures as its content base. The program 
recommends fiction and non-fiction that will expose children to other historical 
periods as well as different parts of the world. In the increasing diversity of 
our nation, as well as the emergence of "the global village," the program helps 
children learn about others through their own eyes. 

Methodology 

Design : Study I (1989-90) utilized a nonequivalent control group design (Cook & 
Campbell, 1979). Pre- and post- writing assessments were administered to 
students in Grades 4-6 between September and May. Mean ratings of pre- and post- 
writing assessments were compared for treatment and comparison groups for each 
of the four thinking skills by grade level, achievement level, and off -task 
responses. Pre- and post- means were also compared for boys and girls, white and 
minority students, and students of different achievement levels within the McRAT 
eroup. Study II (1990-91) was a follow-up of McRAT students in Study I with a 
comparison group of 1st year McRAT students. 

Sample : Six sites in Arkansas were chosen on the basis of their geographic 
diversity and socioeconomic and achievement levels. The sites included rural and 
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urban schools in which there were a range of ethnic compositions including 
majority black and racially balanced. 

The sample for Study I included 234 McRAT students (Grades 4-6) in 16 
classrooms and 106 comparison students in 12 classrooms in six Arkansas schools. 
Schools were selected in terms of geographic diversity and included both McRAT 
and non-McRAT classrooms. All teachers (treatment and comparison) were regular 
classroom teachers, employing similar curricula with the exception of the McRAT 
intervention. Of the 16 McRAT teachers, 11 had received one week of training in 
the previous summer and were beginning their first year with the program while 
five were beginning their second year of the program. The comparison teachers 
had not received training in McRAT. None of the students in either group had 
received McRAT prior to September 1989. 340 students' portfolios were randomly 
selected from each classroom. The percentage of portfolios chosen varied, but 
at least one-third were selected. 

The sample for Study II included 53 second-year McRAT students who were in 
Study I and 53 who were beginning their first year of McRAT instruction. They 
were enrolled in 20 McRAT classrooms. As in Study I, students 1 portfolios were 
randomly selected for inclusion in the study sample. 

Students in both studies were identified by demographics and prior 
achievement. In Study I, approximately 30% of students were minority while in 
Study II, 22% were minority. In Study I,. 25% of the McRAT students were 
identified as educationally disadvantaged and received Chapter 1 services while 
10% of the comparison group were so classified* In Study II, 8% were Chapter 1. 
! 5% of McRAT and comparison students in Study I were classified as "academically 
gifted and talented" while in Study II, 17% were so designated. Samples were 
equal in terms of gender. 

Instrumentation : Four pre- and post- writing assessments were administered to all 
students in the two studies. The rating protocol follows the framework outlined 
by Quellmalz (1985). Essays are scored using five criteria (except Evaluation 
which uses six criteria). Students in each classroom wrote essays (pre- and 
post-) to demonstrate their skills in analysis, comparison, inference 
/interpretation and evaluation. Two independent raters scored each essay with 
the two sets of ratings averaged; scores ranged from 0 to 100 for each essay, 
except Evaluation in Study II where the maximum score was 120. Inter-rater 
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reliability was computed as 0.8 for Study I and 0.95 for Study II. Essays that 
were "off-task" (e.g, not in essay form) were given a score of "0." 
Analysis : Independent -samples hypothesis tests of both pre- and post-tests 
scores were conducted with class mean as the unit of analysis. Other analyses 
were conducted to ascertain the effect of gender, race and prior achievement , as 
well as a separate analysis of students whose pre-test essays were unscoreable 
(0). 

Results 

Table 1 displays the means for all students in Study I whose pre-test 
scores were greater than 0. As can be seen from these statistics, the McRAT group 
initially scored lower than the comparison group on all four essays. The t-tests 
comparing pre-test means revealed statistically significant differences between 
the two groups of classrooms on three of the four measures: Analysis (t=3.2, 
P=.0035); Comparison, (t-0.947, p=.1056); Inference (t=2.2536, p=.0336); and 
Evaluation (t=3.020, p = .0055). 



Table 1 

Mean Ratings of McRAT and Comparison Students 



Skill 


McRAT 


students 




Comparison students 




N 


Pre 


Post 


N 


Pre 


Post 


Analysis 


157 


35.7 


59.3 


99 


44.9 


50.8 


Comparison 


119 


24.4 


51.5 


83 


26.8 


35.5 


Inference 


76 


24.6 


45.5 


65 


31.9 


32.1 


Evaluation 


114 


35.8 


55.3 


99 


45.5 


44.0 



NOTE: These are means for students with scoreable essays for both pre- and post- 
administrations. Results for students who were off-task on the pre-test (0 total 
score) were computed separately. The variations in number of students (N) 
reflect this separation. 

The post-test means demonstrate a clear difference between McRAT and 
comparison students. McRAT classes significantly exceed comparison classes on 
all four thinking skills: Analysis (t=3.27, p = .003); Comparison (t=4.59, p = 
.0001); Inference (t=3.40, p = .0023); and Evaluation (t=3.19; p = .0037). 
Perhaps the most dramatic is Evaluation, where initially, comparison students 
exceeded McRAT students overall by almost ten points. On the post-test, 
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comparison students had remained virtually the same while McRAT students had 
gained over 20 points on the thinking skill that is taught last in the McRAT 
sequence because it encompasses elements of the three others. 

As stated earlier, a considerable proportion (about 30%) of students' pre- 
test essays were "off-task, M and wete given a total score of 0 (zero)* Table 2 
shows that many more McRAT students were included in this group. However on the 
post-test essays, far more comparison students were "off-task" than McRAT 
students. Further examination of the "off -task" group revealed that students 
with one or more "off "task" essays were more likely to be Chapter 1 students. 
The clear implication of this analysis is that educationally disadvantaged 
students are less likely to write connected text without explicit instruction 
such as that provided in KcRAT classrooms. 
Table 2 

Number of McRAT and Comparison Students Off-Task on Pre-test 

PRE-TEST POST-TEST 

Skill McRAT Comparison McRAT Comparison 

Analysis 73 6 1 27 

Comparison 52 13 1 19 

Inference 68 37 10 37 

Evaluation 49 5 7 10 



Table 3 displays the post-test means of McRAT and comparison students who 
were off-task on the pre-test. As can be seen by comparing these means to those 
in Table 1, these students were comparable to their peers after McRAT 
instruction* The meaning of this improvement is illustrated by the case of 
Jonathon, an average fourth grader* His responses to pre- and post-assessments 
for Analysis administered in August and October of 1989 are included in Appendix 
A. Jonathon' s pre-test received a score of 0 (zero) while his post-test was at 
the mean (54.4) for students with pre-tests of 0. 
Table 3 

Post-test Means - McRAT and Comparison Students Off-Task on Pre-test 







McRAT 


Comparison 


Skill 


N 


Mean 


N Mean 


Analysis 


72 


54.4 . 


2 39.0 


Comparison 


51 


43.2 


5 26.0 


Inference 


58 


39.7 


15 32.5 


Evaluation 


45 


49.6 


2 20.0 
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The final analyses in Study I examined the potentially confounding effects 
of students' gender, ethnicity and grade level. No statistically significant 
differences were found for boys and girls, white and minority students . and 
Grades 4, 5, and 6. 

Study II (1990-91) focused on ascertaining the persistence of the effects 
of McRAT over a second year* Table 4 displays the pre* and post-test means for 
students in the second year study who wrote scoreable essays for the pre-tests. 
Table 4 

Comparison of Mean Pre- & Post- Rating of Year I and II McRAT students 



Skill 


1st 


year (N 


=53) 




2nd 


year (N 


=53) 






N 


Pre 


N 


Post 


N 


Pre 


N 


Post 


Analysis 


22 


39.3 


42 


61.8 


48 


52.1 


50 


63.6 


Comparison 


21 


30.6 


41 


62.9 


46 


41.7 


51 


56.5 


Inference 


16 


28.5 


41 


53.0 


44 


42.7 


48 


58.9 


Evaluation 


42 


39.6 


45 


70.1 


49 


47.7 


50 


72.8 



The differences between the means were confirmed by the pre- and post-test 
hypothesis testing. On each of the pre-tests the second year McRAT students 
significantly exceeded students that had not begun the intervention* The first- 
year McRAT pre-tests were very similar to the pre-test scores of both groups in 
Study I. The second year pre-test means were slightly lower than the post-test 
means in Study I, but the gains in writing and thinking skills were still evident 
after summer vacation. The differences between the post-test means were less 
dramatic than in Study I; however, the second-year McRAT students did display a 
slight advantage over their first year counterparts* 

A second indication of the persistence of the effects of McRAT is the small 
number of second year students who were "off-task 1 * on the pre-test measures at 
the beginning of the second year* The pattern of decreased "off-task" responses 
was also evident in Study II. Finally, analysis of the scores of Grade 5 and 6 
students (the two grades enrolling both 1st and second year McRAT students) were 
not differentiated by gender, prior achievement (Chapter 1, "regular" and 
gifted), or race. 

These two studies show that McRAT students demonstrate significant 
increases in higher order thinking* Further, these effects are independent of 
gender, prior achievement level or ethnic background. 
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Discussion 

McRAT is a part of the major shift in states' efforts in educational 
reform, focusing on the "core technology" of the school , classroom instruction. 
It further embodies the movement toward explicit linkages between instructional 
programs and "alternative assessment." Moreover, the assessment techniques used 
by McRAT are notable for their rigorous technical quality. The program is 
consistent with national goals for education in that students clearly demonstrate 
the ability to reason, solve problems, appLy knowledge, write and communicate 
effectively. The program is beneficial to all student groups, including the 
educationally disadvantaged, and it promotes students' understanding of the 
diverse cultural heritage of the United States and the world. 

Por at least the Last decade, both the press and the American public have 
regularly bemoaned the inferiority of American public education and education 
policy -makers have responded with calls for "restructuring" schools. Despite 
many such restructuring efforts (some as well-conceived and implemented as is 
reasonable to expect), there remains ample evidence that regular classroom 
instruction is still dominated by textbooks, teacher lectures, and short answer 
worksheets (Mullis, Owen & Phillips, 1990). In his recent article in Educational 
Researcher . Newmann (1993) argues that "new organizational structures [in 
schools] may be necessary, but [they are] not sufficient" for real changes. He 
further states that restructured schools need to make a commitment to "powerful 
content." The Multicultural Reading and Thinking Program is demonstrably just 
such content. 

For more information about McRAT, readers should contact: 

Janita Hoskyn 
Reading Program Manager 
Arkansas Department of Education 
#4 Capitol Mall 
Little Rock, AR 72201 
Telephone: 501-682-4232 
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